Method and system for asynchronous online distributed problem solving including problems in education, business, finance, and technology

ABSTRACT

Systems and methods facilitating authoring and problem solving by joint contributors working separately but against a common goal. On-line asynchronous distributed authoring and problem solving system, method, and computer program for focusing attention toward particular authoring and problem solving topics using a threaded discussion group and reward matrix. System, method, computer program and computer program product for coordinating the activities of a plurality of people, where the plurality may be any number from two to thousands or more people. Mechanism for directing the attention and focus of large numbers of people who are solving problems using a tree-based problem space, where the tree based problem space may be a virtual problem space. Algorithms and procedures for evaluating nodes in the virtual problem space and assigning values via a pay-off matrix that serves to focus the attention of large numbers of problem solvers. Combination of threaded discussion groups with the pay-off matrix and a variety of algorithms to create useful system for solving multi-level problems leveraging human expertise.

RELATED APPLICATIONS

This application is a divisional of application Ser. No. 09/957,656,filed on Sep. 20, 2001, entitled Method and system For AsynchronousOnline Distributed Problem Solving Including Problems in Education,Business, Finance, and Technology, which claims priority to UnitedStates Provisional Patent Application Ser. No. 60/234,438 filed Sep. 21,2000 entitled Online Distributed Problem Solving System, which is herebyincorporated by reference.

FIELD OF THE INVENTION

This invention pertains generally to systems and methods facilitatingauthoring and problem solving by joint contributors working separatelybut against a common goal, and more particularly to an on-lineasynchronous distributed authoring and problem solving system, method,and computer program for focusing attention toward particular authoringand problem solving topics using a threaded discussion group and rewardmatrix.

BACKGROUND

Heretofore, conventional off-line and on-line problem solving techniqueshave been fraught with limitations. Conventional wisdom and priorresearch on problem solving has traditionally held that that theeffectiveness of group problem solving decreases as the number of peoplein the group increases. Although groups may be useful for brainstorming,for reviewing ideas and documents, and division of labor, it isgenerally agreed that coordination . and communication problems increaseexponentially with the size of the group.

Simply coordinating the calendars of many people working on a commonproblem, set of problems, or project can prove a major obstacle unless(as in the case of the military) there is an absolute authority that candemand people conform to a particular schedule. Simple math shows thatif even if there is a 90% chance that a given meeting time is good foreach particular person involved, there is only about one chance in threethat a particular meeting time will work if ten people are involved(0.9¹⁰=0.34).

What applies to scheduling, also applies to communication. Again, if tenpeople meet, and even if each has a 90% chance of making comments thatkeep the meeting on track and only a 10% chance of making a tangentialcomment that side-tracks the meeting, there is still a two-thirds chancethat the meeting will be side-tracked. As more meeting participants areadded, the situation only gets worse.

With forty-five people participating, again assuming a 10% chance of anoff-track comment or schedule conflict per person, there is more than a99% chance that a scheduled meeting time will not work for someone orthat the meeting will be sidetracked even if the group does manage toget together.

These simple mathematical facts are a major reason why someorganizations try to keep teams small. If large numbers of people areinvolved—it very quickly becomes nearly impossible to get anything done.

On the other hand, teams and even large teams of people do haveadvantages. A primary advantage is summed up by the adage: two heads arebetter than one. With more people, comes more expertise. And research inproblem solving has shown that the availability of relevant expertise isa major factor in coming up with good problem solutions.

Conventional wisdom and prior research on problem solving has thereforeestablished that while small teams may be useful in cases whereadditional relevant expertise is desirable, the coordination andcommunication problems inherent in a concurrent team approach makeslarge teams impractical.

Offline technology such as conference calls on the telephone,video-conferencing, jet travel, and the like, has made it somewhateasier to schedule group work, but as long as the group members must allwork together at the same time, the limitations described above cannotbe avoided.

Conventional on-line problem solving systems and methods also havelimitations. Such online technology systems, ranging from E-mail togroupware and collaborative problem solving systems such as Lotus Notesand other web based systems, represent a great advance over offlineproblem solving technology, primarily because they allow asynchronouswork. For example, with E-mail, group members can exchange ideas withouthaving to both be available for a phone call at the same time. Eachgroup member can work on his or her own schedule—asynchronously.

Unfortunately, even though the scheduling problems are somewhatameliorated by asynchronous offline technologies such as E-mail, theproblem of getting off-track as the number of team members increasesstill exists. In addition, online technologies such as E-mail can easilylead to information overload—something most of us have experienced whenwe come to work and confront dozens (or sometimes even hundreds) ofE-mail messages waiting for us when we return to our workplace.

Structure has been added to E-mail programs to allow users to sortE-mail and categorize it—even to add rules to automatically accept orreject messages. But little has been done with E-mail to facilitategroup problem solving specifically. Those systems that do targetcollaborative work focus on small groups or teams. The softwarebasically attempts to duplicate the same sorts of things that take placein regular conventional offline problem-solving groups—only with theadded capability of allowing members to work asynchronously as well assynchronously, and with the capabilities to exchange documents andinformation online.

E-mail and groupware thus make problem solving somewhat more efficientand easier to conduct across geographically dispersed groups, but littlehas been done to support problem solving by large groups of people.Specifically, substantially all the same communication and coordinationproblems exist when large groups are involved.

Some groupware attempts to overcome some of these coordination problemsvia process or project management techniques that result in verystructured flows on work. For example, with Lotus Notes it is possibleto design document management and workflow solutions that routedocuments from person to person in a very structured way. However, mostNotes applications allow only one person at a time to change thedocuments. If large numbers of people were allowed to simultaneouslymodify Notes documents, chaos would rapidly ensue since there is nocapability in the product to organize revisions by large numbers ofsimultaneous users.

Similarly, the WEBDAV protocol, which represents state-of-the-arttechnology for web-based collaborative authoring—and which has beenadopted by Microsoft, Adobe, and other companies to enable theirexisting applications (for example, Microsoft Word) for collaborativework—allows only one user at a time to modify a document.

Those systems that do allow multiple users to change the same documentsimultaneously are designed for small groups of people working carefullyin different places to avoid the situation of one person undoing someoneelse's revisions.

In short, traditional online groupware applications are currently unableto effectively manage the simultaneous editing of many individuals muchless support more sophisticated types of large-scale problem-solvingefforts.

The Threaded Discussion Group (TDG) represents another standard toolthat has emerged for exchange between many individuals. A threadeddiscussion group is an example of a tree structure. Discussion groupsoftware allows people to post questions on a website. Other people canread these posted questions and respond with answers or relevant ideasof their own.

To keep track of the proliferation of questions, answers, and otherideas that people post, software has been developed that organizespeople's posting according to a tree-structure, where each major topicin the community discussion corresponds to a branch in the tree. As moretopics, also known as threads, are added to the discussion, the treestructure adds more branches to track them. Discussion groups orbulletin boards that make use of threads are called Threaded DiscussionGroups (TDGs).

Recently a variety of websites have come into being that use TDGs,E-mail, and/or other existing online tools to try to provide answers topeople with simple questions about a wide range of subjects. Forexample, the site ASKME.com offers advice on subjects as diverse as UFOsand computer programming. Many other sites exist, all using variants ofthe same or similar technology, that specialize in offering advice indifferent areas, and with different business models—ranging from freeadvice in return for watching advertising to charging a fee forconnecting advice seekers with advice givers, who then solve or attemptto solve the problem or give advice offline using conventionalapproaches.

Many of these online Question and Answer (Q&A) services use TDGs as onemeans for posting advice. A major advantage of this tree structure isthat it organizes the online posts in a hierarchical way, which makes iteasier for people to follow the exchange of ideas. The use oftree-structures as data structures that offer an efficient way oforganizing information is well known in the field of computer science.Since computer scientists invented online bulletin board systems in theearly days of the Internet, it is not surprising that tree-likestructures were used to organize the exchange of ideas.

As Q&A sites are the closest existing technology to an asynchronousonline problem solving system that allows easy access to a wide varietyof experts, the following discussion primarily focuses on the currentlimitations of these systems.

What all existing Q&A sites have in common is that they are able toanswer only relatively simple questions. If a problem requires multiplesteps to solve, or expertise from multiple experts, users are forced tosubmit a series of questions -which is very inefficient and timeconsuming. The net result is that Q&A sites are used for simple, quickanswers. If more complicated problem solving is required, userstypically try to work with experts offline.

In fact, some of the Q&A sites have built their business around thisexisting technology limitation. For example, EXP.COM specializes inmatching people seeking advice with experts who provide bids and then domost of their work offline. This business model is based on the factthat currently there is no good way to solve complicated, multi-stepproblems online—which is why the brokering approach seems attractive.

While TDGs are a powerful technology that facilitates simple online Q&Aservices, known conventional software which displays and organizes theideas of people who post information online, suffer from a number ofdisadvantages that limits the utility and effectiveness of such systemsand methods. Some major disadvantages include the following: (1)Existing online tools and systems that are accessible by large numbersof experts, cannot support complex, multi-step problem solving. (2)Existing online tools and systems that can reach large numbers ofexperts, do a poor job of integrating the work of multipleexperts—especially if these experts do not know each other, and havenever worked together before. (3) Existing online tools and systems allsuffer from the disadvantages that affect all forms of offline andonline problem solving to date, namely that communication andcoordination problems increase and quickly become intractable as thenumber of participants increases.

Some of the reasons that these disadvantages exist in current systems(and especially in TDGs which heretofore represent the best knownapproach to online problem solving that is open to many experts) aresummarized immediately below. First, discussions frequently get offtrack as people express tangential opinions. Second, the amount ofinformation displayed can quickly become overwhelming and take too longto read. Third, people often post repetitive information, which isinefficient and adds to the burden of others trying to find new relevantinformation. Fourth, people with problems have no way of ensuring thatonline experts will check the bulletin board in time for the answers tobe useful to them. Similarly, experts trying to build off of otherexperts' work don't know how long they will have to wait before they canproceed. Fifth, the likelihood of solving a problem tends to decreasemultiplicatively with the number of information exchanges required tosolve the problem because the first four factors each reduce theprobability of successfully completing each information exchange step.Sixth, the likelihood of solving a problem tends to decreasemultiplicatively with the number of experts required to solve theproblem because the first four factors each reduce the probability thata given expert will participate. Seventh, misinformation can be spreadby the system because there is no efficient method for controlling thequality and accuracy of the information posted by experts. Inparticular, rating systems from the question posters, which have beenemployed by some online information exchanges, have limitedeffectiveness at quality control because the very fact that thequestioner is asking a question suggests that the questioner lacks aparticular type of expertise—that, after all, is why she/he is askingthe question. Eighth, a large quantity of time is typically required ofa SYSOP or other human moderator in order to ensure that the local rulesof TDG are followed, and in order to organize and trim the treestructure so that the information exchange remains usable. This list offactors is only exemplary and does not identify all of the reasons forthe failure or limitations of conventional systems and methods.

Slashdot.org is an example of the state-of-the-art technology forreducing irrelevant information on discussion boards. Slashdot asks itstrusted users to rate the postings of other users. Users can thenspecify whether they would like to see all the postings, or only thosepostings with a rating above a specific cut-off number. AlthoughSlashdot (and similar sites such as, for example, Kuro5hin.org andadvogato.org) use ratings to try to filter information, none of thesesites are designed specifically to support problem solving and do notuse ratings as a mechanism for directing the flow of problem solvingactivity.

Therefore, there remains a need for system and method that overcome theproblems and limitations present in conventional approaches.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic illustration showing an embodiment of a problemtree diagram and reward or payoff matrix values which shows severalquestions listed under a general topic in hierarchical tree structureafter a first phase of operation.

FIG. 2 is a diagrammatic illustration showing an embodiment of theproblem tree in FIG. 1 at a later second phase of operation.

FIG. 3 is a diagrammatic illustration showing an embodiment of theproblem tree in FIG. 2 at still a later third phase of operation.

FIG. 4 is a diagrammatic illustration showing embodiments of screendisplays or screen shots during operation of an embodiment of theinvention.

FIG. 5 is a diagrammatic illustration showing an embodiment of a systemaccording to an embodiment of the invention.

FIG. 6 is a diagrammatic illustration showing an embodiment of memory orstorage contents data according to an embodiment of the invention.

FIG. 7 is a diagrammatic illustration showing an embodiment of memory orstorage contents procedures and algorithms according to an embodiment ofthe invention.

SUMMARY

At one level, the invention provide system, method, computer program andcomputer program product for coordinating the activities of a pluralityof people, where the plurality may be any number from two to thousandsor more people. It also provides a mechanism for directing the attentionand focus of large numbers of people who are solving problems using atree-based problem space, where the tree based problem space may be avirtual problem space. The invention further provides algorithms,procedures, and implementations for evaluating nodes in the virtualproblem space and assigning values via a pay-off matrix that serves tofocus the attention of large numbers of problem solvers. Embodiments ofthe invention provide for the use of combinations of threaded discussiongroups with the pay-off matrix and a variety of algorithms from computerscience to create an entirely new and highly useful system for solvingmulti-level problems leveraging human expertise instead of artificialintelligence as has been done previously. Embodiments of the inventionprovide a mechanism and method for drawing the attention of theparticular types of experts to particular places in the virtual problemspace where their expertise is most needed via a database system thatmatches their skills (as specified in the database) to the on-going andever-developing needs for skills of different types as reflected in thevirtual problem space.

The inventive system and method may be applied to solve all manner andtype of problems and in virtually all disciplines that involveintellectual activity and result in the generation of an intellectualwork product, such as a report or recommendation. In a sense anyactivity that would be consider the realm of a consultant or group ofconsultants is ripe for application of the invention. Software design,system design, engineering design and analysis, educational testing andassessment and well as materials preparation, business and financialconsulting, and a great variety of other fields and disciplines benefitfrom the inventive system and method.

In a further aspect, embodiments of the invention provide a method forbuilding-in rules to a threaded discussion group that effectivelyimplement various known search algorithms and allow the implementationof newly created search algorithm, including for example the ability tospecify that no more than a predetermined number of new response can begenerated to any problem before the responses must be evaluated, thebest one chosen, and the process repeated.

In still another aspect, embodiments of the invention provide amechanism for incorporating peer ratings in the context of a threadeddiscussion group or other communication mechanism in which the peerratings lead to a pay-off matrix, and in which the peers themselves areevaluated in terms of their credibility allowing the peer ratings to beweighted by the system as it learns more about the problem solvingeffectiveness of each of the peer-experts.

In an additional aspect, embodiments of the invention provide an abilityto use a threaded discussion group as a virtual problem space, whichmaps the progress of the problem solving effort and provides apermanent, navigable record of the ideas produced together with theevaluation of each idea.

In another aspect, embodiments of the invention provide a system forbringing more minds to focus on a problem simultaneously in acoordinated way than previously existing systems.

In even still another aspect, embodiments of the invention provide asystem that incorporates parameters that allows customers to easilycontrol the problem solving effort by manipulating the values in thepayoff matrix to incent behavior that meets customer needs such asquantity of ideas, quality of ideas and speed with which ideas aregenerated, where as used here, the term “ideas” is a general term meantto encompass intellectual work products of all sorts.

In yet another aspect, embodiments of the invention provide a system foreffectively limiting off-topic posts in a threaded discussion group byvia manipulation of values in a pay-off matrix.

In another aspect, embodiments of the invention provide a system forcombining a threaded discussion group with a synchronized clock cycle sothat there is a periodic cut-off for acceptance of new ideas on aparticular topic which is a parameter that can be controlled by acustomer.

In another aspect, embodiments of the invention provide a system forfiltering content by displaying only those posts that have higher than acertain threshold monetary values as set by the participant and/or thecustomers.

In another aspect, embodiments of the invention provide the ability toreward expert posters based on the degree to which their posts actuallysolved a problem and to use methods of assignment of blame and credit todetermine the value that each post, in a long sequence of posts,contributed towards the ultimate solution of the problem.

In another aspect, embodiments of the invention provide theimplementation of a virtual problem space with search control capabilityusing threaded discussion groups and other techniques.

In still another aspect, embodiments of the invention provide system andmethod for posting a monetary value or other reward or compensation nextto or associated with each post and of providing easy ways for users ofthe system to find the open problems with the most monetary value thatbest fit their expertise

In yet another aspect, embodiments of the invention provide system andmethod that serve various educational, teaching, and learning needsincluding providing capabilities for rapid test item development, rapidscoring of constructed response and essay-types of test items, rapidassociation of educational content with test items/sections, rapiddevelopment of new educational content, and abilities to identify,develop, and share educational practices.

DETAILED DESCRIPTION OF EMBODIMENTS

The inventive Online Distributed Problem Solving (ODPS) system andmethod provides a means to facilitate fast and effective online problemsolving of problems that require multiple steps and/or multiple expertswithout the disadvantages of existing approaches. One significantbenefit of ODPS is that it avoids the problems that force other problemsolving technologies, both online and offline, to limit the number ofexperts or other persons or users working on a problem. Because ODPS canfocus the problem solving capabilities of many experts without sufferingthe usual disadvantages of large groups, ODPS can solve problems fasterand more effectively than any other offline or online system.

The “online” part of ODPS refers primarily to the fact that all theproblem solving work can occur online asynchronously. While face-to-faceor phone contacts between people is possible, and sometimes helpful, itis not necessary for the system to solve complex multi-step problemsand/or problems requiring multiple sources of expertise. This capabilityto solve complex problems online represents a technological leapcompared to existing online Threaded Discussion Groups (TDG) or Questionand Answer (Q&A) systems.

The “distributed” part of ODPS-refers primarily to the fact that theproblem solving work is distributed among experts in such a way that thebest available experts are working on the tasks that best fit theirexpertise. Effective matching of experts to those tasks that fit theirexpertise is a primary reason why ODPS can scale to include largenumbers of experts, without suffering from the communication andcoordination limitations inherent in large groups. The ease and speedwith which tasks can be distributed represents a technological leapcompared to existing online or Offline problem solving technologies.

It will be appreciated that tree-structures have also been used byCognitive Psychologists to model human problem solving behavior. Intheir classic book, Human Problem Solving (1972), Newell and Simondescribe models of human problem solving that use tree-structures whichthey call problem spaces. Theories of human cognition emphasize the rolethat attention plays in successful problem solving.

The inventive system and method recognize that many of the limitationsof existing TDGs were due to the lack of an attentional mechanism.Basically, people cannot solve complex problems using conventionalexisting TDGs alone because there is no good way of maintaining focusedattention of the expert in a multi-step problem. And, when expertisefrom more than one expert is needed, the situation is even worse, almosthopeless in some cases. However, if attention is focused, and if theright sub-problems are brought to the attention of the right experts atthe right time, while preserving a shared representation of the problemthat all experts can refer to, then an efficient and effective system ofonline distributed problem solving is possible.

An embodiment of the inventive ODPS system and method are now describedrelative to a hypothetical problem solving situation. Imagine that acustomer has developed a new software system, named WorldThink, which hewants to test and improve. This customer could use the ODPS system byposting a general topic, together with some more specific questionsrelating to the topic, on TDG. To incent online experts visiting the TDGto answer specific questions, the customer can associate dollar valueswith each of the questions, offering to pay up to the total dollar valuefor each question to experts who provide useful answers. Note thatreferences to threaded discussion groups are merely one example of atype of hierarchical tree based structures that may be used with theinvention and that the invention applies to and may be used with avariety of tree-based structures and techniques. TDGs are convenientlyused in this description as there are many commercial software packagesand Internet web sites that provide or support TDGs.

In FIG. 1 there is shown a tree diagram 100 which shows severalsub-topics are listed under the general topic of. Test and ImproveWorldThink 104, where WorldThink is the name of a hypothetical softwaresystem. Each of these sub-topics represents a branch of the problem treeor virtual problem space (VPS) 101. Experts can help solve the overallproblem of testing and improving the software system called WorldThinkby taking one of the three branches: (i) commenting on any bugs theymight find with the system (Branch a) 112, (ii) commenting on how tomake the system easier to use (Branch b) 114, or (iii) commenting on howto help motivate users of the system (Branch c) 116.

The dollar amounts 120, 122, 124 next to each of these branchesindicates how much money the customer is willing to pay for a highquality ideas that address the topic—help solve the sub-problem—of thatbranch. This assignment of dollar amounts (or other incentives that canbe expressed quantitatively such as points) to problem branches iscalled the payoff matrix 130. The payoff amounts in the payoff matrixrepresent money that has not yet been paid, but is available to be paidfor work that addresses a specific problem or sub-problem. Frequently,problems and sub-problems are expressed as a question.

As an overall problem is split into sub-problems, the total payoffs forall the sub-problems must equal the total payoff for the overallproblem. For example, in FIG. 1, the total payoff for overall problem oftesting and improving WorldThink is $100. This $100 is then spilt amongthe three sub-problems in amounts of $20, $10, and $70—whose sum is theoriginal $100. Amounts showed outside the box or a node are maximumamounts allocated while amounts shown internal to the box of the node isthe amount actually paid out. Note that the amount paid out may be lessthan the maximum. In some embodiments a minimum payout may also beidentified to a node.

In FIG.1, Branch (c) 116 has the highest dollar figure or value amongthe branches available to solve or contribute so experts will be mostmotivated to comment on this topic—as long as they have relevantexpertise in this area. As experts begin to focus attention on thehigh-paying topics, they may post responses.to these topics and/or theymay raise new topics using the standard features of existing bulletinboard or TDG software. Sometime after an expert has posted a response toa question or raised a new topic (which may be a new sub-problem), theresponses and new topics from experts are evaluated, experts are paid,and a new payoff matrix is calculated and posted.

For example, consider the example of FIG. 1, and assume that an expertE1 responds to Branch (c) 116 suggesting a way to motivate users of thesystem by donating money to charity, new Branch (d) 142. Expert E1 alsoproposes a new question, Branch (e) 144, namely: “How do we determinewhich charities are most motivating to people?” An evaluation algorithmruns, evaluates the quality of E1's posts, and determines how much topay E1 for his suggestion of using charity to motivate users. Assume thealgorithm determines E1's suggestion is worth $10. Next, the algorithmthen estimates the worth of E1's new question. Assume the algorithmdetermines E1's new question is worth $5. The algorithm subtracts theamounts due to E1 for his answer and new question ($15) 143 andrecalculates the payoff matrix.

After these calculations, the problem tree 101 and payoff matrix 130would look something like what is depicted in FIG. 2 with added Branch(d) 143 for E1's charity answer 142, and Branch (e) 145 for E1's newquestion 144. Note that the dollar amounts inside the boxes 132, 134represent money paid to experts, while the dollar amounts outside theboxes 120,122, 136 represent the payoff matrix. Note also that thepayoff values for different branches may change or be reassigned overtime to meet customer or other needs.

As a final illustration of how the online problem solving system works,imagine that another Expert (E2) posts a response 148 to the questionabout how to motivate users. E2 suggests a response 148 giving prizes tousers which creates a Branch (f) 149. The algorithm runs, as above, andthe new result is shown in FIG. 3. Note that in this example as moremoney is paid to the experts, the total payoff for answering remainingproblems is reduced. This reflects the fact that progress is being madeon solving the problem, and that the experts are being paid for theirefforts as the problem-solving progresses. If the payoffs drop too low(perhaps even to zero) for a particular topic or for the problem aswhole, focus may be lost unless the customer increases the payoffs byallocating more money to the problem.

Screens shots from an actual prototype system implementing this simpleexample are shown in FIG. 4. These screen shots illustrate the threadednature of the virtual problem space, and show how rewards can be simplyadded to the system by posting dollar values next to each topic. Topicswith higher dollar values tend to attract more problem-solving effortfrom online experts. This simple mechanism is at the heart of the ODPSsystem, as it allows a way of motivating human experts to behave as ifthey were following a wide variety of proven search algorithms.

In FIG. 4A, there is shown a embodiment requesting contributions fortesting and improving the system. It shows the topic, the number ofmessages related to the topic, and the last date and time it wasupdated. In this particular example, features of the ODPS are brieflyexplained and donations to charity for postings are encouraged. In FIG.4B, the overall topic “Test and Improve the System” and the total payoffamount ($500) are listed along with five subtopics are listed along withthe number of messages (Msgs) and the date and time of the last update.A reward or payoff amount of $100 is also identified for contributionsto a listed topic. A button that a contributor may click is alsoidentified for posting a new subtopic. In this example; three subtopicshave received messages or postings and two have not. FIG. 4C illustratesan additional web page showing more detail of the “What could we do tomake the system easier to use?” thread or sub-topic. The sub-topics areeach valued and include $10 for “Could we add better navigationfeatures?”, $10 for “Could we have an auto-numbering feature forposts?”, and $80 for “Could we display more levels of tree all atonce?”.

FIG. 4D shows even further detail for the $80 for “Could we display morelevels of tree all at once?” node, breaking it down into three furtherlevels or branches, including: “Does anyone know of shareware or publicdomain threaded discussion group software?”, “What about building ourown threaded system?”, and “What about a dynamically expandable tree?”Again, provision is made for posting a new subtopic using a clickablebutton, though other means may be provided. FIG. 4E shows a response tothe “What about a dynamically expandable tree?” posting.

Having now described some of the features and procedures of theinventive system and method, an overview of an embodiment of the ODPSmethod is now provided. This overview is followed by a detaileddescription of another embodiment, a discussion of some ways in whichthe inventive system and method overcome disadvantages of conventionalsystems and methods, and a particular exemplary embodiment associatedwith educational and teaching/learning/testing systems.

In one aspect, ODPS is a form of problem solving. Newell & Simon (1972)have shown how all problem solving can be represented as search througha problem space such as though a hierarchical tree structure having aplurality of nodes and branches.

The key to effective problem solving is, according to Newell and Simon,effective search. Effective search, in turn, depends on focusingattention of the problem solver(s) on exploring the part of the problemspace (tree) that has the highest chance of yielding a problem solution.

Embodiments of the ODPS system and method use the existing technology ofTDGs to provide the tree structure that can be searched simultaneouslyby many problem solvers. This online tree structure, implemented by aTDG which resides on a website, is referred to as the Virtual ProblemSpace (VPS). Unlike other forms of software groupware that typicallyrequire small virtual teams to use specialized software in order to makeprogress, the VPS can be accessed by large numbers of online expertssimultaneously, using only a conventional web browser.

To implement search, ODPS focuses and directs the problem solvingefforts of large numbers of experts. By associating rewards (in somequantifiable form, such as for example: money, points, recognition, orthe like) with exploring various branches of the VPS, ODPS focuses anddirects the experts' problem solving efforts. This ability to focus anddirect the attention of the experts is an important foundation for.implementing a wide variety of specific search algorithms.

Search Control Algorithms control the direction of search through theVPS. These algorithms set the rewards and thus effectively control thefocus of many problem solvers as they work in parallel on the problemonline. By adjusting the payoff amounts and other parameters, the SearchControl Algorithms can increase or decrease the likelihood ofinnovation, speed, quality, and other aspects of the problem solutions.

The field of computer science in general, and the sub-fields ofArtificial Intelligence, Game Theory, and Search Algorithms, inparticular, contain many algorithms that may be adapted for use withODPS by programmers and computer scientists skilled in their fields. Forexample, the search control algorithm described above may be consideredto be an enhanced or improved variant of a general search algorithmcalled Best First Search.

Standard textbooks on Artificial Intelligence describe Best FirstSearch, Breadth First Search, Depth First Search, Minimax, and othersearch algorithms—all of which have corresponding implementations inODPS. In fact, any known search algorithm can be adopted to ODPS—whichis one of the powerful strengths of the system.

Standard textbooks and the technical literature describe many well-knownapproaches and methods that can be used to search a problem space. Forexample, the text book, Artificial Intelligence by Elaine Rich(McGraw-Hill 1983, ISBN Number: 0-07-052261-8) herein incorporated byreference describes several of the most well-known approaches andmethods that can be used to search a problem space also referred to as atree structure. These approaches and methods include, but are notlimited to, the following: generate-and-test, hill climbing,breadth-first search, depth-first search, best-first search (includingthe A* algorithm, agendas, and other variations and implementations),problem reduction techniques (including the AO* algorithm),back-propagation techniques, constraint satisfaction techniques(including dependency-directed backtracking), means-ends analysis,branch-and-bound techniques, nearest neighbor algorithm, divide-andconquer techniques, back-chaining techniques, minimax searches (with andwithout cutoffs), as well as the use of heuristic functions of varioussorts. Combinations of one or more of these techniques as well asvariations on these techniques may also be employed.

A point to be understood in applying a search algorithm to ODPS is torealize that the online TDG will implement the problem tree. The rewardsare equivalent or proportional to the result of what is called the valuereturned by the “evaluation function” in computer science. And theexperts themselves (and/or the customer) are the people using theirhuman judgment to conduct the actual evaluation—in most cases. With atree and an evaluation function, search can follow any of the well-knowalgorithms in computer science.

The tremendous power of ODPS derives at least in part from the fact thatODPS can coordinate simultaneously (or nearly simultaneously) thebrainpower of many minds more quickly and effectively than any othermethod hitherto known. Any system that can focus more minds on a problemthat requires multiple steps and/or multiple source of expertise, andthat can do this in a shorter length of time than other systems, willresult in better and faster problem solving than any known method.

A key to maximizing the potential power of the system, in the short run,derives from maximizing the speed with which the evaluation system canproceed, so that the limiting factor becomes the speed with which theindividual human experts can proceed. Once human information processingbecomes the limiting factor, assigning the right human beings to theright parts of the tree becomes critically important. That is, problemsolving works fastest and best if you have experts working in theirareas of expertise. For this reason, the matching of experts to problemsand sub-problems at the beginning and throughout problem solving becomesvery important.

In one aspect, embodiments of the invention provide system, method,computer program and computer program product for coordinating theactivities of a plurality of people, where the plurality may be anynumber from two to thousands or more people.

In another aspect, the invention provides a mechanism for directing theattention and focus of large numbers of people who are solving problemsusing a tree-based problem space, where the tree based problem space maybe a virtual problem space.

In yet another aspect, embodiments of the invention provide specificalgorithms and implementations for evaluating nodes in the virtualproblem space and assigning values via a pay-off matrix that serves tofocus the attention of large numbers of problem solvers.

In still another aspect, embodiments of the invention provide for theuse of combinations of threaded discussion groups with the pay-offmatrix and a variety of algorithms from computer science to create anentirely new and highly useful system for solving multi-level problemsleveraging human expertise instead of artificial intelligence as hasbeen done previously. It will be noted that in the this aspect humanexperts typically provide the problem-solving expertise, but it is alsopossible for the ODPS system to work with a combination or human andnon-human experts (such as for example, Al agents), or with onlynon-human experts.

In a further aspect, embodiments of the invention provide a mechanismfor coordinating the problem solving activity of hundreds or thousandsor more humans as opposed to the limited number of humans who can worksimultaneously to solve problems in a coordinated way using existinggroupware technologies such as for example Lotus Notes.

In yet another aspect, embodiments of the invention provide a mechanismand method for drawing the attention of the particular types of expertsto particular places in the virtual problem space where their expertiseis most needed via a database system that matches their skills (asspecified in the database) to the on-going and ever-developing needs forskills of different types as reflected in the virtual problem space

In a further aspect, embodiments of the invention provide a method forbuilding-in rules to a threaded discussion group that effectivelyimplement various known search algorithms and allow the implementationof newly created search algorithm, including for example the ability tospecify that no more than a predetermined number of new response can begenerated to any problem before the responses must be evaluated, thebest one chosen, and the process repeated.

In still another aspect, embodiments of the invention provide amechanism for incorporating peer ratings in the context of a threadeddiscussion group in which the peer ratings lead to a pay-off matrix, andin which the peers themselves are evaluated in terms of theircredibility allowing the peer ratings to be weighted by the system as itlearns more about the problem solving effectiveness of each of thepeer-experts.

In an additional aspect, embodiments of the invention provide an abilityto use a threaded discussion group as a virtual problem space, whichmaps the progress of the problem solving effort and provides apermanent, navigable record of the ideas produced together with theevaluation of each idea.

In another aspect, embodiments of the invention provide a system forbringing more minds to focus on a problem simultaneously in acoordinated way than previously existing systems.

In even still another aspect, embodiments of the invention provide asystem that incorporates parameters that allows customers to easilycontrol the problem solving effort by manipulating the values in thepayoff matrix to incent behavior that meets customer needs such asquantity of ideas, quality of ideas and speed with which ideas aregenerated. As used here, the term “ideas” is a general term meant toencompass intellectual work products of all sorts.

In yet another aspect, embodiments of the invention provide a system foreffectively limiting off-topic posts in a threaded discussion group byvia manipulation of values in a pay-off matrix.

In another aspect, embodiments of the invention provide a system forcombining a threaded discussion group with a synchronized clock cycle sothat there is a periodic cut-off for acceptance of new ideas on aparticular topic which is a parameter that can be controlled by acustomer.

In another aspect, embodiments of the invention provide a system forfiltering content by displaying only those posts that have higher than acertain threshold monetary values as set by the participant and/or thecustomers.

In another aspect, embodiments of the invention provide the ability toreward expert posters based on the degree to which their posts actuallysolved a problem and to use methods of assignment of blame and credit todetermine the value that each post, in a long sequence of posts,contributed towards the ultimate solution of the problem.

In another aspect, embodiments of the invention provide theimplementation of a virtual problem space with search control capabilityusing threaded discussion groups and other techniques.

In still another aspect, embodiments of the invention provide system andmethod for posting a monetary value or other reward or compensation nextto or associated with each post and of providing easy ways for users ofthe system to find the open problems with the most monetary value thatbest fit their expertise In yet another aspect, embodiments of theinvention provide system and method that serve various educational,teaching, and learning needs including providing capabilities for rapidtest item development, rapid scoring of constructed response andessay-types of test items, rapid association of educational content withtest items/sections, rapid development of new educational content, andabilities to identify, develop, and share educational practices.

The invention provides system, method, computer program and computerprogram product, as well as business operating model for onlinedistributed problem solving capabilities. One particularly usefulembodiment having excellent performance involves four elements, thoughother embodiments may involve only a subset of fewer of these elements.The first step involves specification of the problem to be solved. Thisstep ensures that the problem is well defined and has a clear focus withobjectives and deliverables. The second step involves finding andqualifying the experts that will be working on the problem. This stepincreases the probability that well qualified experts who are availableand able to respond quickly to the problem are included in the problemsolving effort. The third step or element involves use of a virtualproblem space that supports online problem solving by the experts onproblems. This step allows the problem solving to take place online,quickly and effectively. The fourth step or element involvesimplementation of a reward system, combined with search algorithms, thatserves to focus the attention and efforts of online experts as they workon a problem.

With regard to problem specification and expert qualification, ODPS issimilar to other problem solving system in that if the problem is poorlyspecified or the experts are poorly qualified, this will have a negativeimpact on the solutions that are produced. On the other hand, awell-specified problem and well-qualified experts can enhance theresults produced by the system.

ODPS is most effective when the online experts that participate inproblem solving are directed to specific tasks that match their skillsand interests. In particular, for large problem solving efforts, whichinvolve a large virtual problem, a space with many sub-tasks, the moreefficiently that experts can be directed to the relevant sub-tasks, themore efficient the overall problem solving will be.

For purposes of this description, it is assumed that the customerspecifies the problem and initial sub-tasks (if any) by directly usingthe features of the TGD software. This is not a requirement of theinvention but it simplifies the description of the invention by avoidingobscuration of inventive aspects with techniques and procedures ofconventional nature known in the art. It is also assumed that onlineexperts will be notified via email of a URL that corresponds to thetask(s) that best match their qualifications. Again, this assumption ismade for convenience and brevity of description and is not a limitationof the invention itself. Matching algorithms and means for qualifyingexperts, such as administering online questionnaires using questionsthat correspond to features of the customer's problem, are known in theart and not further described here. Software for conducting suchqualification is commercially available from many sources including, forexample from iQ Company, of Capitola, Calif.

Having thus specified the problem and notified the experts, the ODPS canthen produce superior results via a virtual problem space combined withsearch algorithms.

Attention is now directed to detailed description of embodiments of aVirtual Problem Space Component and a Search Control Component. Recallthat the virtual problem space allows experts to actually solve problemsonline efficiently and effectively, and that for problem solving in aVPS to be effective, there must be an algorithm that calculates thepayoff matrix and evaluates the quality of the expert responses andproposed new questions. Embodiments of each of these components aredescribed in turn.

The heart of the ODPS system is the virtual problem space that allowsexperts to actually solve problems online efficiently and effectively.Because, according to the theory of Human Problem Solving, all or nearlyall problem-solving activity can be represented as search in a problemspace, or tree structure, using a tree structure for the virtual problemspace guarantees a general system that is capable of handling any sortof problem. Further, as long as the problem solution can be expressed inthe form of information that can be made available online, and as longas the steps involved in solving the problem involve only manipulationof information that exists in the minds of experts and/or in otheronline sources, then a virtual problem space based on a tree-likestructure can enable any of these sorts of problems to be solved byexchanging ideas online with no in-person, or telephone interactionbeing necessary.

These considerations led to the adoption of TDG technology as thefundamental component of the virtual problem space. TDG systems, asmentioned in the background section and elsewhere in the description,already incorporate a tree structure (though the features of such treestructures have been incomplete exploited). Further, they have softwarefunctions that allow individuals to read what other people have posted,to respond to these posts, and/or to start new threads of thediscussion. Finally, by incorporating universal reference locator (URL)links, TDG systems have the flexibility to include templates, tables,graphics, multimedia presentations, other software programs, and everyother kind of information that is accessible via the worldwide web.Together these capabilities are valuable elements of a virtual problemspace. Note that although TDG may form a valuable basis, the inventionis not limited to TDGs and other mechanisms that provide analogousfeatures may readily be adopted for the inventive system and method.

Recall that some of the features and characteristics that conventionaldiscussion groups and bulletin board systems lack included: (i) aneffective mechanism for directing the attention and efforts of expertsinvolved in a multi-step problem solving activity; (ii) an effectivemethod for filtering information that is irrelevant to problem solving,so that concentration and focus becomes easier; and (iii) an effectivemethod for summarizing the steps taken during problem solving so that aconcise solution can be presented to complex problem.

Beginning with any one of several commercially available or publicdomain TDG software programs, a virtual problem space is created byadding a means of focusing the attention of expert or experts onanswering a particular question or exploring a particular topic. Thismeans can consist of or include any system, procedure, method, ortechnique, that motivates or facilitates motivation of users or expertsto focus more attention or effort on certain branches of the problemtree than on other branches, and even to select one problem over anotherproblem in some instances. The preferred means for ODPS is to add rewardvalues to various branches in the hierarchical tree structure in virtualproblem space. The list (or other data structure) of all or at least asubset of the reward values for all or at least a subset of the variousbranches in the virtual problem space tree is called a reward or payoffmatrix.

The addition of the payoff or reward matrix to the hierarchical treestructure, such as may be implemented in some embodiments with thestructure and method of threaded discussion groups (TDGs) providessignificant capabilities not available in conventional systems ormethods.

The Search Control Component is also important as for problem solving ina VPS to be effective, there must be an algorithm that calculates thepayoff matrix and evaluates the quality of the expert responses andproposed new questions or topics.

The algorithm has specific steps as well as parameters, which allows thealgorithm to be extremely flexible so that it can meet a wide variety ofcustomer needs. One implementation of the algorithm includes thefollowing steps:

-   -   1. If no payoffs have been set for the initial problem and any        sub-problems the customer (or their representative) sets the        payoff amounts. A problem, sub-problem, or topic in the virtual        problem space (VPS) that has a payoff value associated with it        is called a funded topic.    -   2. Intervals (I) are defined either as fixed or variable periods        or time or according to some set of rules or policies. After        every interval (I), check the state of the virtual problem space        to see if there are any new postings—either answers to        questions, new questions (or sub-problems), or both.    -   3. If there have been no new posts, then the state of the        virtual problem space is repeatedly checked (Step 2) until new        postings are detected.    -   4. If a new post exists or is detected, then the quality (Q) of        each new post is determined.    -   5. Determine the payout for each post based on the quality        determination (Q), available funds, and optionally other rules,        policies, or criteria. In one embodiment, the posts are        considered in order of quality. In other embodiments, different        rules may be applied in determining the order for considering        posts—for example to implement various known search algorithms.    -   6. If funds (F) are insufficient at that time to pay for the        post, optionally transfer or attempt to transfer funds from        another funded topic (or other source) in the problem tree. In        one embodiment, attempts to transfer funds desirably begin with        topics that have no posts; and, if after transferring funds from        such non-posted topics there are still insufficient funds,        transfer or attempt to transfer funds from funded topics that        have the lowest quality posts first. Desirably, funds are not        transferred from a topic that has an equal or higher quality        post awaiting payment.    -   7. If there are still insufficient funds after all attempted        transfers, additional funds are requested from the customer.    -   8. Pay the experts for their posts, up to the maximum available        funds.    -   9. Recalculate the payoff matrix taking into account any        transfers of funds between topics and payouts to experts.    -   10. Post the new pay off matrix and payout results. Repeat the        procedure from step 2 until customer or system operator        terminates problem solving.

Some key parameters for this payoff algorithm are: time interval (I),quality (Q), and money or funds (F).

Time and more specifically the length of Time Interval (I), is animportant parameter. The time interval can be set to range from lessthan one second to, minutes, hours, days or more. If the interval isshort, it means that customer wants very rapid responses, and is, or maybe willing, to sacrifice some quality for speed. If the interval islong, typically expected to be on the order of days or weeks, ittypically means the customer wants to see lots of responses from expertsbefore determining the quality of each response. With more responses tochoose from and more input from experts, it is likely the customer willget higher quality responses and more accurate judgments of quality, butit will take more time.

Money or Funds (F) is another important parameter, and specifically, theamount of funds (F) available to payout. In general, the more money thatis made available, the more expert posts that will likely be attracted.Also, it will be recognized that running low on funds means that qualityposts do not get the full funding they deserve and this is de-motivatingto experts. If funds are too low, problem solving may slow or halt, ornot gain the attention of the best experts in the field. In acompetitive environment where top experts have limited time andplentiful problem solving opportunities, a high funding level increasesthe chances of getting the best expertise. It also increases the chancesof getting a rapid response due to the time value of money.

Quality (Q) is a third important parameter, specifically the method ofdetermining the quality (Q) of posts. Quality can be calculated by anyone or more of a number of means. For example, quality may be calculatedor otherwise determined or assessed by human judgment, by automaticalgorithms, or by a combination of human judgment and automaticalgorithms. In many instances, it is desirable to allow ahuman—specifically the customer—to have the final say in qualitydetermination, since quality ultimately translates into customersatisfaction. It will be appreciated that either automated or manualdetermination of quality may be implemented. Furthermore, it is expectedthat as artificial intelligence and neural network, or other techniquesimprove over time, that such automated or autonomous techniques may beused to assist in quality determination. In addition, some automatedquality determination may be used to filter or rate or rank postings toreduce the burden on a final human quality determination or assessment.While a customer may wish to provide a final opinion as to quality,there are other reasons why a customer will probably not wish toexercise this final say over quality except in unusual circumstances.

There are several reasons why the customer may not wish to have thefinal say as to quality. First, usually the customer has posted aproblem because the customer lacks the time or expertise to solve theproblem him or herself. If the customer lacks expertise, then thecustomer may not be the best judge of the actual quality of theposts—especially if they are technical in nature. If the customer lackstime, then the customer probably does not have time to be involved inmaking quality judgments for every cycle of the algorithm—especially ifthe Interval (I) is short. Therefore, the customer will usually wish todelegate the post-by-post quality decisions, and instead exert controlover the problem solving process by a more high-level allocation offunds to generally promising avenues of exploration. Second, it shouldbe noted that involving humans directly in making every qualityjudgment, would limit the speed of the problem solving system.

In addition to techniques that would involve direct customer or otherhuman involvement in determining quality, there are several methods ofdetermining the Quality of a post without direct involvement from thecustomer. One of the most effective methods is to leverage the expertiseof the online experts echo are posting to the system. In one embodiment,this online expert based quality assessment method can be implemented asfollows:

(i) Experts post responses to questions and/or raise new questions asdescribed herein elsewhere in this description.

(ii) Experts are free to read some or desirably all the posts of some ordesirably all other experts, and can rate these posts on a qualityscale. This quality scale may for example include or consist of one ormore dimensions depending on how much detail and what type of is desiredor required. It will be appreciated by those workers having ordinaryskill in the art in light of the description provided here, that thereare tradeoffs as to the degree of detail provided in quality assessmentsor ratings. For example, provision of more detailed ratings require moretime, but many existing rating scales (such as those used by ebay,ASKME.com, or the like) show it is possible to construct a simple scalethat provides useful information without being burdensome to the raters.

(iii) In one embodiment of the expert based quality rating system andmethod, averages or other statistically based derivations of the ratingsof all the experts for each post that has been rated are computed orotherwise determined. These measures derived from the raw expertratings) can be straight averages, weighted averages, or otherstatistically or deterministically derived measures. If weightedaverages were used, one method would be to weight the rating for thepost by the overall rating of the expert who is doing the rating. Thisapproach assumes that experts themselves have ratings that are composedof several dimensions such as customer satisfaction, or the like, asdiscussed above under the topic of qualifying experts.

(iv) Based on the processed quality rating (such as for example theaverage or weighted average rating) of each post, the inventive systemand method can consider the posts in order of quality as described inthe payoff matrix algorithm.

(v) In one embodiment, additional steps may optionally be taken after aproblem has been solved to provide additional information as to thequality of postings made by experts or other contributors. For example,in one embodiment, after a problem has been solved, the posts should bere-evaluated (either by an algorithm, human(s), or a combination of oneor more humans assisted by algorithm based automated analysis) todetermine which posts made the greatest contribution to the problemsolution. Then those experts who rated these posts highly should gaincredibility as peer raters while those who rated these posts poorlyshould lose credibility as peer-raters. In subsequent ratings, theopinions of peer raters with higher credibility should be given moreweight than those peer raters having lesser credibility. Desirably,peer-raters would provide expertise in particular problem areas and thecredibility of a rater would be tied to their demonstrated area ofexpertise.

(vi) In one embodiment, usually at the customer's option, a certain sumof money or funds (or other non-monetary award) can be reserved to bepaid out as bonuses to those experts whose ideas were determined to -bemost relevant (after the fact) to actually solving the problem.

It will be appreciated by those workers having ordinary skill in the artin light of the description provided herein that the payoff method andalgorithm described herein above is only one of many different types ofalgorithms that can be used to determine and/or influence how to focusattention in online problem solving.

The particular algorithm which will perform the best, will in generaldepend upon the goals and needs of the customer, to a large extent. Thecustomer may be interested in obtaining problem solutions under anynumber of different scenarios, for example: quickly, by a certain date,inexpensively, with extremely high quality, or according to othercriteria or factors. Depending on the relative importance of each ofthese factors, parameters in the algorithm (such as the interval of timebefore all posts are evaluated, or total rewards offered) can be tunedto optimize the fit for the customer.

Having described features and characteristics of one embodiment of theinventive system and method for the general problem, attention is nowdirected to an embodiment of the invention as applied to teaching andlearning in the educational field or market.

Applications of ODPS to the Education Market

In the description of embodiments of the invention so far, thedescription has explained the general ODPS algorithm and illustrated tohow to implement ODPS using a prototype system that implemented the ODPSsystem for solving generic problems whose solution can be represented byan idea expressed in a paragraph of text. However it may be noted thatthe same ODPS algorithms can be included in more specific types ofproblem solving systems. This section describes five such specific ODPSsub-systems—all targeted to the education market.

State, local, and federal governments in the United States are veryconcerned about improving student achievement, especially in theelementary or kindergarten through high-school (K-12) grades. Thisconcern is shared by countries outside the United States as well as bycompanies that serve country, state, and school administrators,educators, parents, and students.

What gets measured gets done, so student testing is becoming anincreasingly important component of initiatives to improve studentachievement. This focus on testing raises many problems, which can bemore efficiently solved by embodiments of the inventive ODPS system andmethod than by currently existing systems and methods. Some of the keyissues requiring or benefiting from solution include: (1) How can themany new test items that will be needed to accommodate the increasedfocus on student testing be rapidly developed? (2) How can constructedresponse (“essay type”) questions be rapidly scored to allow the fastestpossible turn-around of test scores to administrators, teachers,parents, and students? (3) How can existing educational content belinked to the areas where tests show students need the most help? (4)How can new educational content that addresses the areas where testsshow students need the most help be rapidly developed? (5) How can bestpractices in education be identified, developed, and shared so as toimprove or facilitate improvement in student achievement?

Rapid Test Item Development

Development and validation of educational test items conventionallytakes several years and requires careful research studies and analysisto produce a relatively small number of“high stakes” test items.High-stakes test items include for example, the items used as part ofscholastic achievement tests, high school exit exams, college andgraduate school entrance exams, and similar standardized tests whichmeasure student achievement. The long period of development maygenerally constrain the ability to improve education.

Embodiments of the inventive ODPS system and method adapted foreducational or other test item development are capable of producinglarge quantities of high-quality “low stakes” test items which could beused immediately for “low stakes” tests, and/or fed into existingvalidation processes to- produce a much larger quantity of “high stakes”test items. Low-stakes test items include for example, the studyquestions found at the end of textbooks, test items used by individualteachers in specific schools, or self-study test items used as homeworkor aids to learning specific material, and other items which have notbeen statistically normalized so as to compare the achievement ofstudents from different populations.

One embodiment of an ODPS system and method that achieve these resultsis implemented as now described. Other embodiments that include some ofthese features or variations on these features may also or alternativelyimplemented.

A secure web-based virtual problem space (VPS) is established usingknown security measures, such as security implemented using SSL, usernames and passwords, or the like and a threaded discussion group (TDG)or equivalent.

In this educational environment, the top-level problem in the VPS is todevelop test items. Sub or lower level problems within the problem treeinclude developing specific types of test items in specific contentareas for specific grade levels.

The payoff matrix reflects the needs of the customer at any given timeand may be updated or modified as the needs change. For example, if dueto a new contract there is a sudden need for test items covering 11^(th)grade American History, the payoff for developing items of this type isincreased until enough high-quality items have been generated.

For simple items that have a known format, the solution to thesub-problem of generating a needed item could be accomplished in asingle post. The post would link to existing forms or templates, such asfor example to existing web-based item templates, which the onlinesubject matter expert would complete. For more complicated items, ortest sections that are composed of several items, several individualsmight respond and create a team solution in several problem-solving (oritem creation) steps. Again, some of these steps could bepre-determined. In one situation, one step might be to create the textof the question and response choices, while another step might be tocreate an accompanying graphic or other media or multi-media materials.

The quality of all solutions (and/or solution pieces) is subject to peerreview and/or customer review using a variation of the general ODPSpayoff algorithm described elsewhere herein. The customer wouldfrequently be the final arbiter, deciding whether to accept or rejectitems (solutions) and adjusting the credibility of the peer expertsaccording to their track record of recommending or rating items that aremostly accepted or mostly rejected. As a result of the feedback frompeer review, problem solvers may revise their items until they pass peermuster and then submit them to the customer for final review andpayment, if they are found acceptable.

A database stores information about each registered problem solver suchas his or her areas of expertise, past performance as a problem solver,and credibility as a peer rater, and/or other relevant information. Whena problem solver logs-on, the ODPS system identifies matches between theproblem solver's qualifications and open problems. Techniques forconstructing such a database-matching algorithm are known in the art andnot described in greater detail here. Open problems available forsolution are listed for problem solvers in order according to criteriathat the problem solver might specify, for example in order ofdecreasing payoff, increasing payoff, most recent problems to be solvedfirst, or other order.

Item templates and/or other authoring tools or software or computerprograms or code (e.g. applets) that problem solvers might use for thespecific task of authoring items would be provided via links from thespecific task threads in the TDG.

Once items have been accepted by the customer, the customer is able totransfer these items into. a customer database of approved items. Thecustomer database may be a database separate from the ODPS sub-systemfor rapid item development or may comprise storage associated with theODPS system. Techniques for transferring approved items from a completeditem template, linked in a TDG, to a customer database are known in theart and not described in greater detail herein.

Embodiments of the ODPS sub-system for rapid item development providesignificant benefits as compared to conventional systems and methods.Significant benefits of the ODPS sub-system for rapid item developmentinclude: (1) Just-in-time item development based on the immediate ornear-term needs of the customer. (2) More items developed more quicklythan is possible using conventional approaches to item development dueto the large number of people who could be working simultaneously onitem development. (3) High levels of quality control (as a result of thepeer-rater system) that imposes a minimal burden on the customer. (4)Access to a far greater range of potential item experts than would bepossible using conventional approaches. (5) Seven day per week,twenty-four hour per day (24×7) item development capability withworldwide experts contributing or using the system. (6) Built-introubleshooting and item revision capabilities due to the peer reviewsystem combined with the TDG discussion capabilities and the ODPSproblem solving structure.

Rapid Scoring of Constructed Response Test Items

Currently it takes at least twice as long for testing companies to scoreessay type questions which are also referred to as constructed responsequestions, as it does to score multiple-choice questions. The additionaltime required reflects the fact that human scorers typically read eachconstructed response item and then assign a score whereas computers canscore multiple-choice items.

Although recently the Educational Testing Service has experimented withcomputer scoring of constructed response items, such systems aregenerally acknowledged to be inferior to human scoring. Computerizedscoring systems generally must pick out key words or phrases in essayanswers and assign points based on the presence of absence of these keywords or phrases. In contrast, human scorers can understand the entireessay and thus can score on overall meaning—even if a particular word orphrase was not used.

Since it is generally acknowledged that the purpose of the tests is toassess knowledge—as opposed to the ability to use a word or phrase thata computer scoring system might be looking for—computerized scoring is aless than ideal solution to the problem of trying to score constructedresponse items more rapidly. (In the future, as techniques for scoringsuch constructed response items improve, the use of computer or othernon-human scoring may be expected to increase.)

An ODPS system and method for scoring of constructed response items mayreduce the current time required for scoring constructed response items,for example depending upon question type, improvements of 50% and moremay be expected in eliminating the delay that is now caused by this typeof test item without resorting to computer scoring.

An ODPS system for rapid scoring of constructed response type items isnow described.

A secure web-based VPS is established using known security measures(e.g. SSL, user names & passwords) and a TDG.

In this embodiment, the top-level problem in the VPS is to scoreconstructed response test items. Sub-problems can include scoringspecific types of test items in specific content areas for specificgrade levels.

The reward or payoff matrix is constructed and updated so that itreflects the scoring needs of the customer at any given time. Forexample, if the current need is to score essays covering the War of 1812(in a test on American History) the payoff for scoring these essays areincreased until all of the essays have been scored.

For most constructed response items, the solution to the sub-problem ofscoring the item against pre-existing criteria may be accomplished in asingle post. In other situations, multiple posts may be needed or if notneeded, desired so as to receive additional information. The post wouldlink to existing web-based scoring templates, which the human scorerwill complete. It will be appreciated that for this embodiment of theinventive system and method, “problem solver” equates to “scorer” sincethe problem to be solved is one of scoring exams. Scoring of otherauthored materials may be accomplished in analogous manner. Forcomplicated items, or test sections that are composed of several items,one or more individual problem solvers (scorers) may respond and scorein several problem-solving (scoring) steps. Again, these steps may be toapply pre-determined scoring criteria. For example, one step in thescoring procedure may be to score the text of the question while anotherstep may be to score a required accompanying drawing.

The quality of all solutions (scores) is subject to review, in oneembodiment to peer review and/or to customer review. In some situations,it will be desirable to have multiple individuals score the same item,but this is not a requirement. If the two independent scorers agree on ascore it is passed on to the customer. If there is disagreement, asub-problem of resolving the disagreement may be established and the twoscorers can for example, use the capabilities of the TDG to discusstheir differences and reach consensus on a score. One of their optionsis to bring in additional scorers as may be necessary until a consensus,a majority, an opinion, or some other predetermined policy or rulesabout a correct or acceptable score is reached.

In some embodiments, the customer is the final arbiter, breaking scoringdeadlocks and adjusting the credibility of the scorers according totheir scoring track records or history. For example, a scorer whoconsistently scores differently (and incorrectly or with some bias inthe eyes of the customer) from his or her peers would have a lowercredibility rating, and might ultimately be eliminated from the pool ofpotential scorers altogether.

A database stores information about each registered scorer such as hisor her areas of expertise, past performance as a scorer, credibility asa scorer, and any other information that may be useful for scoring. Whena problem solver logs-on, the ODPS system identifies matches between thescorer's qualifications and open items to be scored. Database-matchingalgorithms and systems are known in the art and not described in greaterdetail herein. The open items to be scored are listed for scorers inorder of decreasing payoff, or according to other criteria that thescorer might specify. In one embodiment, open items to be scored arelisted in order of most recent items first, while in another embodimentthey may be listed in order of the oldest items first.

Scoring templates, guidelines and/or other scoring tools or programs(such as for example in the form of computer software code or applets)that scorers might use for the specific task of scoring items would beprovided via links from the specific task threads in the TDG.

Once scored items have been approved by the customer (or other approvalauthority), the customer (or other party) transfers these items into acustomer database of approved items. The customer database may be thesame or separate from the ODPS sub-system for scoring. Systems andprocedures for transferring such scored items are known in the art andnot described in greater detail here.

The inventive system and method for the ODPS sub-system for rapidconstructed response item scoring provide may benefits and advantagesover conventional systems and methods, including but not limited to: (i)higher quality scoring than could be achieved by conventional computerbased techniques due to the superior comprehension ability of humans andthe ability for discussion and consensus scoring in ambiguous orborderline cases; (ii) faster scoring than is conventionally possibledue to the simultaneous work of many human scorers using a secure,web-enabled ODPS scoring system; and (iii) ability of the customer toset the priority of items to be scored and to control variables likespeed and quality of scoring by manipulating the payoff matrix. Thislater feature allows flexibility in the amount of attention given toeach item; and more important or more complicated items can be givenmore scrutiny and/or scrutinized by more than one scorer.

Linking Educational Content to Assessment Items

The purpose of student testing is ultimately to improve studentachievement. Therefore a component of testing is to reveal wherestudents need additional instruction. However the once tests scores areknown, it is often a difficult and time consuming task to determinespecifically which instructional materials or information would be mosthelpful to individual students. Because teachers typically must dividetheir time among many students, they are often forced by timeconstraints to choose instructional material geared towards the averagestudent rather than tailored to each student's individual instructionalneeds. Furthermore, conventionally decisions may be made for an entireschool district rather than on the basis of an individual school orclass within a school.

Recognizing this problem, companies that produce student tests areinterested in linking scores on their tests to specific educationalcontent that might help students improve in the areas where the testsreveal they need help.

One proven method of matching educational content to weak test scores isto “tag” the educational content in some way so as to indicate that aparticular chunk of educational content is relevant to a particular testitem. A hyperlink (or other reference) can be constructed to theappropriate tags, or a search program can locate and displayappropriately tagged information. For example, if a student misses ageometry problem on a general math test, a relevant passage from ageometry textbook could be recommended to the student, provided that thegeometry passage was “tagged” as being related to the item that thestudent missed.

Unfortunately this approach of linking educational content to test itemsvia tags (henceforth called “linking”) is labor-intensive. Automatedtagging approaches that search for key words in content are possible,but generally trade-off quality for speed. That is, you can tag lots ofeducational content using an automated system, but the material that yourecommend is often not really what the student needs.

An ODPS system for linking educational content to specific test itemsvia tags could reduce the current time required for such linking itemsby 50% or more, while still providing links to highly relevanteducational content.

One embodiment of the ODPS system to link test items with relevantinstructional content is implemented as follows:

A secure web-based VPS is established using known security measures (forexample SSL, user names and passwords) and a TDG. The security measuresare intended to allow identification and/or control as to who providesproblem solutions and who is entitled to receive payment.

The top-level problem in the VPS is to link test items with relevantinstructional content. Sub-problems can include linking specific typesof test items in specific content areas for specific grade levels,schools, subjects, or the like.

The payoff matrix reflects the needs of the customer at any given time.For example, if the current need is to link instructional covering theWar of 1812 to items on an American History test, then the payoff forlinking this content is increased until all of the test items haveinstructional links. For most linking tasks, the solution to thesub-problem of linking the item to instructional content may beaccomplished in a single post. The post would link to existing web-basedlinking templates, which the problem solver (“linker”) would use to taginstructional content relevant to the test item. Available instructionalcontent could be displayed and/or the linker could specify otherrelevant content that he or she might know due to his or her expertise.For complicated items, or test sections that are composed of severaltest items, one or more individual linkers might respond and link inseveral problem-solving steps. Again, these steps might be to selectfrom pre-existing educational content or to suggest new content.Depending on the amount of instructional content desired for each testitem, several linkers might tag content that they found mostappropriate. The relevance value of those items tagged by severallinkers could be increased, so that the most popular content wasrecommended first to students missing the test item. Teams of linkers,using the discussion features of the TDG may be able to generate morerelevant content than linkers working individually, due to the primingof ideas that happens during a discussion. Therefore, linkers (and/orthe customer) may find it advantageous to have team discussing andgenerating content tags for the same item.

The quality of the tags is subject to peer review and customer review.As mentioned above, it may be desirable to have multiple linkers tag forthe same test item. If independent linkers disagree about whethercertain content should be tagged, a sub-problem of resolving thedisagreement is established and the two linkers can use the capabilitiesof the TDG to discuss their differences and reach consensus on whatcontent to tag. One of their options is to bring in additional linkersas may be necessary until a consensus, or at least a majority, opinionabout the correct content to be tagged is reached. The customer would bethe final arbiter, breaking deadlocks and adjusting the credibility ofthe linkers according to their track records. For example, a linker whoconsistently links incorrectly (in the eyes of the customer and/or hisor her peers) would have a lower credibility rating, and mightultimately be eliminated from the pool of potential linkers altogether.

A database stores information about each registered linker such as hisor her areas of expertise, past performance as a linker, and credibilityas a linker. When a linker logs on, the ODPS system identifies matchesbetween the linkier's qualifications and open test items to be linked toeducational content. Constructing such a database-matching algorithm issomething within the ordinary skill in the art and not described infurther detail here. The open items to be linked would then be listedfor linkers in order of decreasing payoff or according to other criteriathat the linker might specify, such as for example most recent itemsfirst.

Linking templates, guidelines, lists of available content to be linkedand/or other linking tools or programs that linkers might use for thespecific task of linking items would be provided via links from thespecific task threads in the TDG.

Once linked items had been approved by the customer or other approvalauthority, the customer would be able to transfer these items into acustomer database of approved items, separate from the ODPS sub-systemfor linking.

It will be appreciated that embodiments of the invention providenumerous advantages and benefits as compared to conventional systems,methods, and techniques, including but not limited to: (i) providinghigher-quality links to educational material than could be achieved by acomputer due to the superior comprehension ability of humans and theability for discussion and consensus scoring in ambiguous or borderlinecases; (ii) providing faster linking than is currently possible due tothe simultaneous work of many human linkers using a secure, web-enabledODPS linking system; (iii) providing an ability of the customer to setthe priority of items to be linked and to control variables like speedand quality of linking by manipulating the payoff matrix. This allowsflexibility in the amount of attention given to each item. Moreimportant, or more complicated, items can be given more attention andlinked to more types of educational content (perhaps by more linkers).

Rapid Development of New Educational Content

Developing instructional content is also a difficult and time-consumingtask that is typically performed primarily by one (or a small number of)content author(s); Currently many educators are developing very similarinstruction for the same or similar classes. Assembling new content fromalready developed pieces and allowing authors to focus on developingcontent in the areas where they have the most expertise would result inmuch more efficient creation of new educational content.

An ODPS system supporting the rapid development of educational contentgreatly improves the efficiency of the development process and resultsin a higher quality educational product.

One embodiment of the ODPS system to provide this functionality isimplemented as now described. A secure web-based VPS is establishedusing known security measures and a TDG or equivalent. In thisembodiment, the top-level problem in the VPS is to develop a specificpiece of instructional content such as a course or lecture for aparticular educational use. Sub-problems include developing sub-piecesof the content—for example individual overhead slides in a lecture orlectures in an overall course. There will be sub-sub-problems andsub-sub-sub-problems down to the level of granularity where a problemcan be solved by one action.

The payoff matrix for this embodiment reflects the needs of the customerat any given time. For example, if the current need is to developinstruction covering the War of 1812, then the payoff for developingthis content is increased until content has been developed. Within theoverall task of developing a complex piece of content (e.g. a course),the payoff matrix can reflect ongoing priorities for development ofvarious sub-pieces. Experts can be attracted where they are needed andwhen they are needed by the changing payoff values that can be set bythe customer, the primary content developer, an algorithm, or acombination of humans and algorithms as discussed previously.

For the simplest content development tasks, the solution to thesub-problem of developing a unit of content may occur in a single post.The post would link to existing web-based development templates, whichthe problem solver (“developer”) would use to author the piece ofcontent. For complicated content chunks, one or more individualdevelopers might respond and develop in several development steps. Thesesteps might include authoring text, creating graphics, findingreferences, preparing syllabi or any of a number of content developmentsteps familiar to authors skilled in the art of developing educationalcontent. Teams of developers, using.the. discussion features of the TDGmay be able to generate more relevant content than developers workingindividually, due to the sharing of ideas that happens during adiscussion. Therefore, developers (and/or the customer) may find itadvantageous to have teams discussing and developing content pieces thatwill be assembled into a larger overall unit of instructional content.

The quality of all content developed is subject to peer review andcustomer review. In some embodiments, it may be desirable to havemultiple developers develop alternative versions of the same content. Ifindependent developers disagree about whether certain content piecesshould be included in the larger product, they can use the capabilitiesof the TDG to discuss their differences and reach consensus on whatcontent pieces to include. One of their options is to bring inadditional developers as may be necessary until a consensus, or at leasta majority, opinion about the correct content to be included is reached.The customer may be the final arbiter, breaking deadlocks and adjustingthe credibility ratings of the developers according to theirperformance. For example, a developer who consistently develops poorquality content (in the eyes of the customer and/or his or her peers)would have a lower credibility rating, and might ultimately beeliminated from the pool of potential developers altogether.

A database stores information about each registered developer such ashis or her areas of expertise, past performance as a developer, andcredibility as a developer. When a developer logs-on, the ODPS systemidentifies matches between the developer's qualifications andeducational content that needs to be developed. The open developmentjobs are then listed for developers according to a set of rules,policies, preferences, or the like.

Development templates, guidelines, list of available content pieces,and/or other development tools or programs (e.g. applets) thatdevelopers might use for the specific task of developing content wouldbe provided via links from the specific task threads in the TDG.

Once linked items had been approved by the customer, the customer wouldbe able to transfer these items into a customer database of approveditems, separate from the ODPS sub-system for linking.

It will be appreciated that making a contribution to a problem solutionis not limited only to providing a text or symbolic contribution as inWritten prose. Rather contribution may take virtually any form andtemplates and tools may be provided directly, or linked or identified insome manner such as by a hyperlink associated with the topic or node.For example, if the problem statement required developing some contentitem the system may display or link to additional instructive orsupporting material to assist the developer in developing content thatsatisfies a required form. Alternatively, such information may beprovided after content provided in some free-form has been selected fromalternative contributions.

Embodiments of the invention of the ODPS sub-system for rapiddevelopment of educational content provide numerous advantages overconventional systems and methods, including: (i) Higher-quality contentthan could be achieved by individual authors working alone or in smallgroups due to the ability to match experts to the specific areas wherethey are most qualified to develop content pieces and to the ability fordiscussion and quality control using multiple distributed experts; (ii)faster development than is currently possible due to the simultaneouswork of many human developers using a secure, web-enabled ODPSdevelopment system; (iii) an ability of the customer to set the priorityof items to be developed and to control variables like speed and qualityof development by manipulating the payoff matrix. This allowsflexibility in the amount of attention given to each development task.More important, or more complicated, tasks are given more attention andmay be worked on by more developers.

Rapid Identification, Development and Sharing of Educational BestPractices

Educators worldwide face many of the same problems. An efficient meansof identifying, developing, and sharing the best solutions toeducational problems would be of great value. Currently, many discussiongroups, email lists, bulletin board systems, and libraries exist—allfocused on best practices in education. However, none of these existingsystems is designed to specifically support problem solving.

For example, in a typical discussion group, educators can raise problemsand post possible solution ideas, but, as mentioned earlier, the depthof problem solving rarely reaches more than three or four levels becausethere is no good mechanism in the discussion group to sustain and directproblem solving through the many steps that may be needed to reach aspecific solution.

Embodiments of the inventive ODPS system and method are adapted to helpidentify, develop, and share educational best practices more effectivelythan existing systems because it could draw upon worldwide expertise andbecause it possesses the capability to direct and sustain attentionthrough the many steps that may be needed to adapt that expertise (e.g.a best practice) to a particular need.

These embodiments of the ODPS system and method may be implemented asnow described. A secure web-based VPS is established using knownsecurity measures and a threaded discussion group (TDG) or otherequivalent mechanism.

The top-level problem in the VPS is to develop a solution to aparticular educational problem—for example increase the number of thirdgraders who can read at grade level over the next year. Sub-goals andsub-problems are set by the participants as necessary during the courseof problem solving.

The payoff matrix reflects the needs of the customer and the state ofthe problem solving progress at any given time. For example, if theparticipants rate choosing the right instructional materials as one ofthe most promising next steps to solve the overall problem of increasingthird grade literacy, then this sub-problem will have a high payoffassociated with it. This high payoff will in turn attract more expertsto this particular branch of the VPS and focus attention and efforthere. Thus experts can be attracted where they are needed, and when theyare needed, by the changing payoff values that can be set by thecustomer, the primary content developer, an algorithm, or a combinationof humans and algorithms as discussed previously.

For the simplest educational problems, the solution may consist of asingle post, for example a pointer to the URL of an already existingbest practice in education. However for more complex problems, severalindividuals might respond, generating alternative next steps and workingtogether to solve the problem in several problem solving steps. Thesesteps might include a wide variety of activities including postingideas, conducting research on the web, referencing other materials, orcontacting other experts. Teams of problem solvers, using theasynchronous discussion features of the TDG will be able to generatemore relevant solution ideas than developers working individually, dueto the sharing of ideas that happens during a discussion and the factthat many minds are working on the problem with different expertise.

The quality of all solutions (and solution ideas) is subject to peerreview and customer review. Typically multiple problem solvers willdevelop alternative next steps at each stage of the problem solvingeffort. The payoff matrix is adjusted to focus attention on one or moreof these next steps as the most promising for further exploration.Independent problem solvers can debate the merits of each other'ssolutions ideas using the capabilities of the TDG. The customer is thefinal arbiter of the quality of the solution produced. Problem solverswho consistently suggest poorly rated solution ideas (in the eyes of thecustomer and/or his or her peers) would have a lower credibility rating,and might ultimately be eliminated from the pool of potential problemsolvers altogether.

A database stores information about each registered problem solver suchas his or her areas of expertise, past performance, and credibility.When a problem solver logs-on, the ODPS system identifies matchesbetween the problem solver's qualifications and the problem that needsto be solved. The open problems are then listed for problem solvers inorder of decreasing payoff or according to other criteria that theproblem solver might specify (such as most recent items first).

Problem solving templates, guidelines, list of available resources andcontacts, and/or other problem solving tools or programs (e.g. applets)that problem solvers might use for the specific task of identifying,developing, and sharing best practices would be provided via links fromthe specific task threads in the TDG. For example, with respect tosharing the solutions, one sub-problem would be to index the solutionsso that other educators could easily access the solutions once they weremoved into the sharing database as described below.

Once solutions had been approved by the customer, the customer would beable to transfer these solutions into a “sharing database” of approvedsolutions, separate from the ODPS sub-system for the identification anddevelopment of best practices. The sharing database of solvededucational problems would be made accessible to educators.

The benefits and advantages provided by this embodiment of the inventionof the ODPS sub-system for identification, development, and sharing ofeducational best practices include, but are not limited to, thefollowing: (i) identification and development of higher-quality bestpractices than could be achieved by individuals working alone or insmall groups due to the ability to match experts to the specificproblems, due to the world-wide reach of the system, and due to the peerreview problem-solving process which ensures high levels of quality;(ii) faster identification and development than is currently possibledue to the simultaneous work of many human problem solvers using asecure, web-enabled ODPS system; (iii) an ability of the customer to setthe priority of problems to be solved and to control variables likespeed and quality of solutions by manipulating the payoff matrix. Thisallows flexibility in the amount of attention given to each problemtask. More important, or more complicated, tasks are given moreattention and may be worked on by more problem solvers.

Embodiment of an ODPS System

With reference to FIG. 5 there is now described an embodiment of theinventive system for implementing an ODPS system. In this embodiment, anODPS server 202 is coupled to the Internet 204 in conventional manner.Experts 206 may also connect to the ODPS server 202 from a clientcomputer 208 via the Internet. Internet connectivity using conventionalbrowsers are preferred as it permits a maximum number of contributors orexperts to access available open problems and post contributions orsolutions to such problems without requiring specialized applicationsoftware. It is also possible for the ODPS server to send specializedsoftware to any contributor or expert on a client device 208. Customers210, such as entities that post a problem for solution may also connectto the ODPS server via the Internet, though direct dial up or any otherconnection may be provided. In one embodiment, the customer operates theODPS server and therefore separate access is not required. The entiresystem may alternatively be operated as an intranet, such as for examplewhere a company or other organization operates it for their own internalpurposes. One example would be for a Department of Defense or othersecure application.

Most likely the customer would connect to the system in a similar way tothe experts—i.e. via browser, only using different software with strongsecurity. This time, instead of the e-business software paying out moneyto experts for their work, the e-business software would be billingcustomers for the work. Again, the customers would be able to accesstheir account via browser, but the account management software maytypically reside on the ODPS server(s).

In one embodiment, the ODPS server(s) 202 provide a powerful set of webservers, connected to databases that run: (1) the threaded discussiongroup software/databases (TDGs) that contain the ongoing problem solvingwork; (2) the e-business software that handles payments of experts andbilling of customers; (3) the ODPS problem database and expertisedatabase and the software that matches experts to problems; (4) theprocedures and software that runs the reward or payoff matrix and allthe other functions of the ODPS system; (5) the authoring tools and/orother specific problem solving tools and templates that might be neededby experts (accessible via a browser); and (6) operating systems andother software that the above software might rely upon.

Each of ODPS server 202, expert client computer 208, and customercomputer 212 is of conventional type having a processor or CPU 214,memory 216 coupled with the processor or CPU 214, and possibly variousinput/output and peripheral devices 218, such as for example displayscreen, keyboard, mouse or pointing device, hard disk drive or othermass storage devices, printers, and the like, as are known in the art.ODPS server 202 will also have mass storage such as may be provided by ahard disk drive or array of hard disk drives, either of which may bepart of the server or provided as network attached storage. It will alsobe appreciated that although one ODPS server is specifically shown,multiple servers may conveniently be provided for additional capacity orredundancy, and such may be collocated or geographically dispersed.

Each of the server 202, expert 206, and customer 210 computers will havecertain data and procedures appropriate for authoring content (problemsor contributions to problem solutions) as well as administrative tasks.These data and procedures may typically be stored on hard disk drives orother local storage as is known in the art.

The ODPS server 202 may for example provide procedures in the form ofcomputer software programs for an operating system 220, TDG software222, Authoring software applications 224, e-business applicationsoftware 226 including for example account management software 228. Theserver 202 may also include a browser so that it may interact as aclient when required for normal access or for trouble shooting anddebugging.

It is noted that authoring procedures, tools, and /or software mayreside on the client machines and/or on the server, and that when neededmay be communicated from the server to the client computers used byexperts when needed via browser using known techniques, such as forexample, using SERVLET technology and methods. Alternatively, anauthoring tool (for example, an applet) may be downloaded to theexpert's browser. Either or both implementations may be used. Alsoe-business software of some kind would desirably and advantageously beprovided on the server to pay, reward, or otherwise compensate theexperts for their work. This software advantageously includes accountmanagement software so that experts could review their accounts, see howmuch they have earned, look at their credibility ratings, input orrevise their qualifications, and other administrative functions. In oneembodiment, all of these functions would reside on the server with theexperts accessing via a browser. However, other embodiments may providethat certain of these components would be downloaded to the experts'browsers, such as through the use of applets, for reasons of efficiency.

Each expert client 208 computer 206 which may take the form of any typeof conventional computer or information appliance may also store dataand procedures in the form of computer software programs. In generalsuch expert machine 208 will provide an operating system 232, a webbrowser such as internet explorer or other network browser, authoringapplications or tools such as conventional word processing applicationsor specialized applications made available for interacting with the ODPSsystem 200.

Each customer 210 computer 212 may be similar to or the same as anyother expert client machine, particularly where any customer specificapplications are provided on the ODPS server rather than on the customermachine. Any customer specific applications may alternatively beprovided on the customer machine 212 if desired.

Most likely the customer would connect to the system in a similar way tothe experts—i.e. via browser, only possibly using different softwarewith strong or stronger security. This time, instead of the e-businesssoftware paying out money to experts for their work, the e-businesssoftware would be billing customers for the work. Again, the customerswould be able to access their account via browser, but the accountmanagement software may typically reside on the ODPS server(s).

Those workers having ordinary skill in the art will appreciate thatsoftware and/or data may be stored in one or both of OPDS server 202 andclient (expert or customer) 208, 212 depending upon the implementationdesired. With reference to FIG. 6, elements of data that areadvantageously provided and stored for at least one of the aforedescribed ODPS system applications are now described. It is noted thatonly a subset of these data and procedures are needed for particularones of the applications described herein.

In one embodiment, the ODPS server 202 would store data either in memoryduring use of the particular data or more usually in a mass storagedevice such as a hard disk drive or disk drive array. Data components240 may include: a problem tree 242; reward or payoff matrix and matrixvalues 244; registered experts 246 (including for example one or more ofregistered problem solvers, registered scorers, registered contentlinkers, registered content developers, or other registered experts forother or different fields); templates, forms, or tools 248; expertmatching procedure data 250 for matching experts or contributors withdifferent open problem sets or other available tasks (such as forexample scorer qualification matching procedures); reward or payoffmatrix data 252; item or file transfer data 254; expert and customeraccount information 256; problem description and status data 258;problem solution data 260; work-in-progress associated with problem treenodes 262; expert qualification data 264; and other data that may beneeded or desired to support one of more of the inventive ODPS features.

With reference to FIG. 7, there are illustrated some exemplaryprocedures and algorithms that may be used for or in conjunction withthe inventive system and method. These procedures may advantageously beimplemented as computer programs and computer program products. Suchcomputer program products may be stored on tangible media such asfloppy-disk or CD-ROM or communicated electronically and stored by thesender or receiver within solid state memory, hard disk drive, or othermeans.

Procedures and algorithms 270 implemented in computer program software,firmware, or other means may for example include: an operating system272, various application programs 274, a TDG procedure application 276,an ODPS payoff procedure 278, an expert qualification to problemmatching procedure 280, an expert rating procedure 282, a reward ofpayout matrix update procedure 284, and item or file transfer procedure286, one or more search procedures 288, and other procedures 290 as maybe desired or required to implement particular features or capabilitiesof the OPDS system and method.

It is noted that the various procedures shown in the diagram includemany procedures for a variety of different applications and that for anyparticular application many of these procedures would not be used andare optional. For example, in an ODPS system designed and implementedfor developing test items procedures for scoring test items or tools forother specific types of problem-solving would not be required.

Other Exemplary Applications

It will be appreciated that the inventive ODPS is applicable to solvinga great variety of problems, and that in particular it is applicable tosolving problems in the educational field such as the development ofeducational content and assessment test items, scoring assessment itemsand resolving scoring conflicts, solving specific educational problemsrelated to improving student achievement and identifying, developing,and sharing educational best practice, as described above.

Other areas of applicability are now described by way of example but notby way of limitation. ODPS may be used for solving traditionalconsulting problems such as management and strategic consultingproblems, logistical problems, marketing problems, financial problems,human resource problems, manufacturing problems, and other similar ofproblems whose solutions consists of ideas, reports, or otherdeliverables that are knowledge products.

It may also be applied to solving humanitarian and charitable problemsthat may require the coordination of large numbers of people, ideas, andresources.

Another applicable field is to address design problems such as thedesign of web sites, software, technology, or other products whosedesign can be broken into parts that can be worked on by more than oneperson simultaneously.

Military and government applications also benefit from the inventiveODPS system and method including solving military and organizationalstrategy problems, problems related to improving efficiency, problemsrelated to large-scale coordination of personnel and resources,simulation problems, problems related to war games, and other similarproblems.

Games and simulations such as chess, checkers, Go, and other commongames as well as multi-player computer-based games, and other types ofgames or entertainment that may require input from or the coordinationof large numbers of players, are another field of applicability.

ODPS may also be applied to information sharing and communicationproblems and situation involving large numbers of people who desire toget the right information at the right level of detail to meet specificindividual informational needs and overall goals. Decision supportproblems represent a specific class of information sharing andcommunication problems that could especially benefit from the ODPSapproach.

Modeling problems in which economic models, environmental models, orother sorts of models requiring input and/or coordination of largenumbers of people may be involved are yet another area of applicability.

Having described some of the disadvantages and limitations ofconventional problem solving approaches, as well as various embodimentsof the invention it will be apparent that the invention overcomes manyof the problems or limitations. A brief explanation is set forth belowas to how embodiments and/or features of the inventive ODPS system,method, and computer program and computer program product overcomeparticular ones of such conventional problems and limitations.

The first major disadvantages of existing TDGs and other online tools isthat even if they are accessible by large numbers of experts, theycannot readily support complex, multi-step problem solving. Inparticular, discussions in existing TDG systems rarely go more thanthree or four levels deep into the tree. An example of three levels is:A first person raise a topic, a second person comment on the firstperson's topic, and then the first person comment on the second person'scomment. Theoretically there could be comments on each other commentsindefinitely. However in practice, discussions rarely get more thanthree or four deep because the posters of the topic or comment loseinterest or someone posts a new top-level comment.

In other words, it is difficult for existing TDGs to sustain the focusedattention necessary to explore a topic further than three or four levelsdeep. The relatively shallow exploration of topics that occurs withtraditional TDGs means that TDGs have traditionally been used to handleonly very simple problems (that is simple question and answer typeproblems) that require only about two or three levels of depth pertopic. Unfortunately this excludes a large number of problems thatrequire deeper exploration in order to find a good or complete solution.

ODPS overcomes this limitation incorporating a mechanism for directingand focusing the attention of experts and others attempting to provideproblem solutions. In a regular TDG, some experts may respond to aquestion, but the probability that you would get continued responses toa series of questions decreases with the number of questions (i.e. withthe required depth of topic exploration). With ODPS, because experts arefocused on payoffs (see description of payoffs elsewhere in thisdocument) for responding to various questions, a customer posing theproblem for solution can keep expert attention for as many solutionsteps as are needed. Not only that, customers can switch expertattention easily and quickly by simply changing the parameters orstructure of a payoff matrix associated with questions posted to theTDG.

The second major disadvantage of existing TDGs is that they do a poorjob of integrating the work of multiple experts—especially if theseexperts do not know each other, and have never worked together before.TDGs are typically a “free-for-all” where anybody can bring up any topicthat interests him or her. Some TDGs may concentrate in a narrower areabut still questions may be rather freely posed. This lack ofgoal-directed focus means that it is extremely difficult to coordinateefforts of multiple experts. If the experts don't know each other, thenthey share even less of a common context, and coordination becomes thatmuch more difficult.

ODPS provides a clear and simple method for telling experts what isimportant in a problem. As experts answer questions and raise newquestions, the system provides feedback about which answers and newquestions merit reward. This feedback, is a very simple, elegant, andeffective way to encourage experts to work together along lines that thesystem has determined are most likely to lead to a problem solution.Experts need only “follow the money” or other reward and the searchalgorithm that controls the reward or payoff matrix values ensures thatprogress is being made. Integration, summarization, and other tasks thatarise during the course of problem solving can be incented by the systemand accomplished by the experts at the appropriate times. The capabilityof ODPS to direct experts to problems that match their expertise furtherincreases the efficiency of integrating work from several experts.

A third major limitation of existing TDGs and other forms of offline andonline problem solving is that communication and coordination problemsincrease and quickly become intractable as the number of participantsincreases. ODPS solves the problem of communication and coordination ofmany experts in a way that is similar to the way a free market works.For example, in the stock market, no computer program or organizationalstructure could possible coordinate all the research, buying, andselling activities that take place every day by huge numbers of people.Instead, the market simply sets a price, and the individuals are free torespond to the price as they see fit. If more choose to respond byselling, the price drops. If more choose to buy, the price increasesuntil equilibrium is reached. Many complex activities are driven by avery simple mechanism.

Similarly, with ODPS, the system does not force any one expert torespond to a problem at a particular time. But because large number ofexperts are involved, and because the incentives for working onparticular problem solving paths are changing dynamically based on theactions of all the other experts (e.g. the progress being made on theproblem and the ongoing evaluation of this progress), statisticallyspeaking, the right expertise ends up going where it is needed. at anymoment in time to make progress on the problem. This approach to problemsolving bypasses the usual problems of coordinating specific experts,because each individual is free to contribute as he or she feels at anymoment. The net result is equivalent to what would be achieved if largenumbers of experts worked together in a coordinated fashion—but none ofthe headaches involved with trying to coordinate the behavior ofindividual people are involved. The key point to remember is that ODPScares about getting the right ideas to the place they are needed to makeprogress on the problem. ODPS does not care which expert provides theidea, and thus ODPS is not bound by the same limitations as conventionalapproaches to problem solving which quickly get mired in the inherentdifficulties of trying to coordinate communication and work betweenspecific individuals working on a team. ODPS is a market approach—not ateam approach—to problem solving.

A brief description as to how the inventive ODPS system and methodovercomes specific disadvantages or limitations of existing TDGs orother problem solving tools now follows. (The traditional limitationsare in italics and an embodiment of the ODPS approach in normalnon-italic font.)

(1) Problem: Discussions frequently get off track as people expresstangential opinions. Solution: With ODPS, tangential opinions andoff-track comments are not rewarded, so experts tend to ignore them andfocus on goal-oriented discussion, which is rewarded.

(2) Problem: The amount of information displayed can quickly becomeoverwhelming and takes too long to read. With ODPS, experts can focus onthose problem posts that have been validated by receiving rewards. Iffurther filtering is required, ODPS supports displaying only posts abovea certain dollar value, or only from highly rated experts. ODPS alsodirects experts to the portion of the virtual problem space that bestmatches their expertise, thus presenting them with the most relevantinformation first.

(3) Problem: People often post repetitive information, which isinefficient and adds to the burden of others trying to find new relevantinformation. Solution: ODPS does not pay for repetitive information, sothere is an incentive to read what others have posted, and avoidduplication. In the long run, this reduces the amount of clutter for allexperts and makes problem solving more efficient.

(4) Problem: People with problems have no way of ensuring that onlineexperts will check the bulletin board in time for the answers to beuseful to them. Similarly, experts trying to build off of other experts'work don 't know how long they will have to wait before they canproceed. Solution: ODPS does not enforce response from specific expertswithin specific timeframes. This approach leads to coordinationdifficulties and scales very poorly. Rather, ODPS relies on marketfactors to incent fast response. Specifically, if two experts respondwith essentially the same information, the first expert to respond (asmeasured by the time and date stamp of the post) is rewarded, while theresponse from the second expert is considered a repetitive post. Withlarge numbers of experts, customers can have very good control over thepace of problem solving by manipulating parameters of the ODPS systemsuch as the value of the rewards. Higher rewards lead to faster responsetimes as experts rush to post their responses before some other expertscoops them. For a problem with above average rewards, experts know thatthere will be above average response times, which encourages them tocheck in with the system often if they wish to share in the rewards.More modest rewards tend to lead to slower response times as a naturalconsequence of the fact that experts are less willing to adapt theirschedules to work on the problem. Thus no overt control need be exerted,but market forces act very effectively to ensure rapid response ifrewards are sufficient.

(5) Problem: The likelihood of solving a problem tends to decreasemultiplicatively with the number of information exchanges required tosolve the problem. Solution: The problems of distracting posts andoff-track ideas do not divert the attention of ODPS experts becauserewards keep them on track, even over the course of many problem solvingsteps and topics. Thus, the likelihood of solving problems is a functionof the rewards for each step, not the number of steps involved.

(6) Problem. The likelihood of solving a problem tends to decreasemultiplicatively with the number of experts required to solve theproblem. Solution: Because ODPS does not have the coordination problemstypically involved with large numbers of experts, it actually becomeseasier, rather than harder, to solve problems the more people that areinvolved. This counter-intuitive result, which runs against conventionalwisdom concerning group problem solving, reflects the fact that moreexperts increases the likelihood that one will be available with thecorrect piece of information at any given moment in time. Since theactivities of the experts are governed by market forces, rather than byovert control strategies, ODPS does not pay the traditional performancepenalty for large group size that all conventional approaches pay. Moreexperts mean more expertise, delivered more quickly with no exponentialincrease in coordination difficulties.

(7) Problem: Misinformation can be spread by the system because there isno efficient method for controlling the quality and accuracy of theinformation posted by experts. In particular, rating systems from thequestion posters, which have been employed by some online informationexchanges, have limited effectiveness at quality control because thevery fact that the questioner is asking a question suggests that thequestioner lacks a particular type of expertise—that, after all, is whyshe or he is asking the question. Solution: ODPS uses a peer ratingsystem that asks experts to evaluate their own posts, as well as theposts of other experts. Even though some experts may inflate ratings oftheir own contributions, on average, the ratings that most experts agreeare superior will receive a higher rating. This method, combined withsafeguards designed to avoid cheating and to ensure that the customer'svalues and needs are being met, allows problem solving progress to beevaluated by those who have the expertise needed to make good judgments.Unlike slashdot.org or other discussion board systems with peer-ratingsystems, ODPS systems have clear criteria for what constitute helpfulposts—namely those posts that solve—or help solve—the problem arehelpful. Those that don't contribute to the solution are noise. Using“problem solution” as the criteria of excellence allows the ratersthemselves to be evaluated based on how highly they rated the posts thatended up solving the problem. Poor quality posts, mis-information, andexperts that do a poor job of rating the posts of others can be quicklyrecognized and eliminated using this approach.

(8) Problem: A large quantity of time is typically required of a SYSOPor other human moderator in order to ensure that the local rules of TDGare followed, and in order to organize and trim the tree structure sothat the information exchange remains usable. Solution: Experts who postclear and concise information that is accurate, timely, and of highquality, earn the most positive ratings from their peers and the highestrewards from the customer. Thus, the-market forces, combined with thegood judgment of the expert community, act together to organize and triminformation. Organized information is simply more highly rated andtherefore more valuable. Again, the market, and the collective actionsof many individuals replace the work that is done in conventionalsystems by a single individual. This collective rating approach has beenvalidated by slashdot.org for reducing “noise” in discussion systems,but prior to ODPS, it has not been used to guide search through manylevels of a tree structure in order to solve problems.

The inventive ODPS therefore provides superior performance as comparedto existing systems, technologies, and methods.

One of the most powerful features of the online distributed problemsolving system is that many minds can work simultaneously, in parallelon a problem, and that problem solving can proceed theoretically as fastas new posts are received. This means that the system can solve problemsmany times faster than real-world experts.

For example, if ten experts spend 15 minutes posting to the system in anhour, this is equivalent to 2.5 experts. Now it is possible for 2.5experts to work together offline for an hour and probably they wouldachieve comparable if not superior results to ODPS. But now imagine that10,000 experts spend 15 minutes posting to the system. This isequivalent to the brainpower of 2,500 experts concentrating on a problemfor an hour. It is impossible to coordinate this many experts offline ina single hour, but online, ODPS can do it.

Even assuming a 90% loss of efficiency (which is much higher than mightreasonably be expected), the 10,000 experts would still translate intothe equivalent of 250 experts focused intently on solving a problem foran hour, working together, building off each other's ideas. This is asituation that is simply impossible in the offline world. This meansthat in terms of expertise delivered per unit of time, it will be inpractical terms, virtually impossible to beat ODPS by any known offlinemethod. Since competitive advantage, innovation, and in fact mostsources of new value are now created via intellectual activities, andsince the people who arrive at the new ideas first have a tremendousadvantage, ODPS becomes very valuable.

The inventive ODPS system and method therefore represent a new way ofconducting online problem solving of problems that require multiplesteps and/or multiple experts, though it may clearly also be used withsimple problems as well. A key feature is the combination a treestructure, such as is readily available in TDG software, with amechanism to direct search through the tree—the payoff matrix and searchalgorithms discussed elsewhere herein. When search through a virtualproblem space is combined with a system for matching many expertssimultaneously to problems and sub-problems where they have expertise,rapid and effective problem solving surpassing anything previouslyobtainable via any hitherto known system is possible. The disadvantagesthat stem from coordinating large numbers of individuals are replaced bythe advantages that accrue from a market system, free from exponentiallyincreasing coordination and communication problems, that performs betterand better as more experts are involved.

While the present invention has been described with reference to a fewspecific embodiments and examples, the description and the particularembodiments described are illustrative of the invention and is not to beconstrued as limiting the invention. Various modifications may occur tothose skilled in the art without departing from the true spirit andscope of the invention as defined by the description and the appendedclaims. All patents and publications referenced herein are herebyincorporated by reference.

1. An asynchronous online distributed problem solving system comprising:means for solving problems online that require multiple steps ormultiple sources of expertise without resort to offline communication orcoordination; means for focusing the attention and problem solvingefforts of experts on specific tasks within a virtual problem spacedefined in part by a hierarchical problem tree including sub-problemtasks and a problem reward matrix by manipulating rewards associatedwith various sub-problem tasks; and means for distributing tasks toexperts in the virtual problem space in a way that allows substantiallyunlimited numbers of experts to work simultaneously on a problem.
 2. Amethod for directing the attention of contributors to particular topicsamong a plurality of topics in a problem tree structure, the methodcharacterized in that reward values are associated with each of theplurality of topics.
 3. The method in claim 2, wherein the reward valuesare updated at time intervals to maintain the attention of contributorsof a particular topic or to alter the attention of contributors todifferent topics.
 4. The method in claim 2, wherein the problem treestructure includes a threaded discussion group structure.
 5. A methodfor solving a complex problem, the characterized by dynamicallymodifying the complex problem formulation during its solution inresponse to at least one received problem solution contributionincluding having a contributor formulate and present at least one newsub-problem to be solved in response to problem solution contributionssubmitted.
 6. A system comprising: a server having a processor, memorycoupled to the processor, and storage; means for solving problems onlinethat require multiple steps or multiple sources of expertise withoutresort to offline communication or coordination; means for focusing theattention and problem solving efforts of experts on specific taskswithin a virtual problem space defined in part by a hierarchical problemtree including sub-problem tasks and a problem reward matrix bymanipulating rewards associated with various sub-problem tasks; andmeans for distributing tasks to experts in the virtual problem space ina way that allows substantially unlimited numbers of experts to worksimultaneously on a problem.
 7. A method for solving a complex problem,said method comprising: formulating a complex problem to be solved as aproblem tree having a plurality of problem nodes and problem branchescoupling said problem nodes, wherein each of said plurality of problemnodes comprises a sub-problem , and wherein contributions to the subproblem contribute to solution of the complex problem; associating areward matrix with said problem tree having reward values identifiedwith at least some of said sub-problems; receiving a problem solutioncontribution to a particular one of the sub-problems from a particularproblem solving contributor; and dynamically modifying the complexproblem formulation during its solution in response to at least onereceived problem solution contribution including having a contributorformulate and present at least one new sub problem to be solved inresponse to problem solution contributions submitted.